2000 character limit reached
Segmentation, Classification and Interpretation of Breast Cancer Medical Images using Human-in-the-Loop Machine Learning (2403.20112v1)
Published 29 Mar 2024 in cs.CV and cs.LG
Abstract: This paper explores the application of Human-in-the-Loop (HITL) strategies in training machine learning models in the medical domain. In this case a doctor-in-the-loop approach is proposed to leverage human expertise in dealing with large and complex data. Specifically, the paper deals with the integration of genomic data and Whole Slide Imaging (WSI) analysis of breast cancer. Three different tasks were developed: segmentation of histopathological images, classification of this images regarding the genomic subtype of the cancer and, finally, interpretation of the machine learning results. The involvement of a pathologist helped us to develop a better segmentation model and to enhance the explainatory capabilities of the models, but the classification results were suboptimal, highlighting the limitations of this approach: despite involving human experts, complex domains can still pose challenges, and a HITL approach may not always be effective.
- Siegel, R.L., Giaquinto, A.N., Jemal, A.: Cancer statistics, 2024. CA: A Cancer Journal for Clinicians 74(1), 12–49 (2024) https://doi.org/10.3322/caac.21820 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21820 Dizon and Kamal [2024] Dizon, D.S., Kamal, A.H.: Cancer statistics 2024: All hands on deck. CA: A Cancer Journal for Clinicians 74(1), 8–9 (2024) https://doi.org/10.3322/caac.21824 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21824 Giaquinto et al. [2022] Giaquinto, A.N., Sung, H., Miller, K.D., Kramer, J.L., Newman, L.A., Minihan, A., Jemal, A., Siegel, R.L.: Breast cancer statistics, 2022. CA: A Cancer Journal for Clinicians 72(6), 524–541 (2022) https://doi.org/10.3322/caac.21754 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21754 Parker et al. [2009] Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Dizon, D.S., Kamal, A.H.: Cancer statistics 2024: All hands on deck. CA: A Cancer Journal for Clinicians 74(1), 8–9 (2024) https://doi.org/10.3322/caac.21824 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21824 Giaquinto et al. [2022] Giaquinto, A.N., Sung, H., Miller, K.D., Kramer, J.L., Newman, L.A., Minihan, A., Jemal, A., Siegel, R.L.: Breast cancer statistics, 2022. CA: A Cancer Journal for Clinicians 72(6), 524–541 (2022) https://doi.org/10.3322/caac.21754 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21754 Parker et al. [2009] Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Giaquinto, A.N., Sung, H., Miller, K.D., Kramer, J.L., Newman, L.A., Minihan, A., Jemal, A., Siegel, R.L.: Breast cancer statistics, 2022. CA: A Cancer Journal for Clinicians 72(6), 524–541 (2022) https://doi.org/10.3322/caac.21754 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21754 Parker et al. [2009] Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Dizon, D.S., Kamal, A.H.: Cancer statistics 2024: All hands on deck. CA: A Cancer Journal for Clinicians 74(1), 8–9 (2024) https://doi.org/10.3322/caac.21824 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21824 Giaquinto et al. [2022] Giaquinto, A.N., Sung, H., Miller, K.D., Kramer, J.L., Newman, L.A., Minihan, A., Jemal, A., Siegel, R.L.: Breast cancer statistics, 2022. CA: A Cancer Journal for Clinicians 72(6), 524–541 (2022) https://doi.org/10.3322/caac.21754 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21754 Parker et al. [2009] Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Giaquinto, A.N., Sung, H., Miller, K.D., Kramer, J.L., Newman, L.A., Minihan, A., Jemal, A., Siegel, R.L.: Breast cancer statistics, 2022. CA: A Cancer Journal for Clinicians 72(6), 524–541 (2022) https://doi.org/10.3322/caac.21754 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21754 Parker et al. [2009] Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Giaquinto, A.N., Sung, H., Miller, K.D., Kramer, J.L., Newman, L.A., Minihan, A., Jemal, A., Siegel, R.L.: Breast cancer statistics, 2022. CA: A Cancer Journal for Clinicians 72(6), 524–541 (2022) https://doi.org/10.3322/caac.21754 https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21754 Parker et al. [2009] Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Parker, J.S., Mullins, M., Cheang, M.C.U., Leung, S., Voduc, D., Vickery, T., Davies, S., Fauron, C., He, X., Hu, Z., Quackenbush, J.F., Stijleman, I.J., Palazzo, J., Marron, J.S., Nobel, A.B., Mardis, E., Nielsen, T.O., Ellis, M.J., Perou, C.M., Bernard, P.S.: Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27(8), 1160–1167 (2009) https://doi.org/10.1200/JCO.2008.18.1370 https://doi.org/10.1200/JCO.2008.18.1370. PMID: 19204204 Pascual et al. [2019] Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Pascual, T., Martin, M., Fernández-Martínez, A., Paré, L., Alba, E., Rodríguez-Lescure, A., Perrone, G., Cortés, J., Morales, S., Lluch, A., Urruticoechea, A., González-Farré, B., Galván, P., Jares, P., Rodriguez, A., Chic, N., Righi, D., Cejalvo, J.M., Tonini, G., Adamo, B., Vidal, M., Villagrasa, P., Muñoz, M., Prat, A.: A pathology-based combined model to identify pam50 non-luminal intrinsic disease in hormone receptor-positive her2-negative breast cancer. Frontiers in Oncology 9 (2019) https://doi.org/10.3389/fonc.2019.00303 Gurcan et al. [2009] Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: A review. IEEE Reviews in Biomedical Engineering 2, 147–171 (2009) https://doi.org/10.1109/RBME.2009.2034865 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., Fernández-Leal, A.: Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 3005, 3054 (2023) https://doi.org/10.1007/s10462-022-10246-w Kosaraju et al. [2022] Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Kosaraju, S., Park, J., Lee, H., Yang, J.W., Kang, M.: Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports 12(1), 19075 (2022) https://doi.org/10.1038/s41598-022-23166-0 Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nature Reviews Clinical Oncology 16(11), 703–715 (2019) https://doi.org/10.1038/s41571-019-0252-y Su et al. [2022] Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Su, A., Lee, H., Tan, X., Suarez, C.J., Andor, N., Nguyen, Q., Ji, H.P.: A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images. NPJ Precis. Oncol. 6(1), 14 (2022) Rosai [2007] Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Rosai, J.: Why microscopy will remain a cornerstone of surgical pathology. Lab. Invest. 87(5), 403–408 (2007) van der Laak et al. [2021] Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021) Bera et al. [2019] Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Bera, K., Schalper, K.A., Rimm, D.L., Velcheti, V., Madabhushi, A.: Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 16(11), 703–715 (2019) Schneider et al. [2022] Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Schneider, L., Laiouar-Pedari, S., Kuntz, S., Krieghoff-Henning, E., Hekler, A., Kather, J.N., Gaiser, T., Fröhling, S., Brinker, T.J.: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. European Journal of Cancer 160, 80–91 (2022) https://doi.org/10.1016/j.ejca.2021.10.007 Schettini et al. [2022] Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Schettini, F., Brasó-Maristany, F., Kuderer, N.M., Prat, A.: A perspective on the development and lack of interchangeability of the breast cancer intrinsic subtypes. NPJ Breast Cancer 8(1), 85 (2022) https://doi.org/10.1038/s41523-022-00451-9 Cancer Genome Atlas Research Network et al. [2013] Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Cancer Genome Atlas Research Network, Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A., Ellrott, K., Shmulevich, I., Sander, C., Stuart, J.M.: The cancer genome atlas pan-cancer analysis project. Nat Genet 45(10), 1113–1120 (2013) https://doi.org/10.1038/ng.2764 Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26(6), 1045–1057 (2013) https://doi.org/10.1007/s10278-013-9622-7 Mosqueira-Rey et al. [2024] Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Mosqueira-Rey, E., Hernández-Pereira, E., Bobes-Bascarán, J., Alonso-Ríos, D., Pérez-Sánchez, A., Fernández-Leal, A., Moret-Bonillo, V., Vidal-Ínsua, Y., Vázquez-Rivera, F.: Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach. Neural Computing and Applications 36(5), 2597–2616 (2024) https://doi.org/10.1007/s00521-023-09197-2 Anderson et al. [2016] Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Anderson, M.R., Antenucci, D., Cafarella, M.J.: Runtime support for human-in-the-loop feature engineering system. IEEE Data Eng. Bull. 39(4), 62–84 (2016) Gkorou et al. [2020] Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Gkorou, D., Larranaga, M., Ypma, A., Hasibi, F., Wijk, R.J.: Get a human-in-the-loop: Feature engineering via interactive visualizations. In: Proceedings of the Workshop on Interactive Adaptive Learning Co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2020), vol. 2660. CEUR Workshop Proceedings, ??? (2020). https://ceur-ws.org/Vol-2660/ialatecml_shortpaper4.pdf Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. ICML ’09, pp. 41–48. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380 . https://dl.acm.org/doi/10.1145/1553374.1553380 Holmberg et al. [2020] Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Holmberg, L., Davidsson, P., Linde, P.: A feature space focus in machine teaching. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–2 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156175 . http://mau.diva-portal.org/smash/get/diva2:1428195/FULLTEXT01.pdf Settles [2009] Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences (2009). https://minds.wisconsin.edu/handle/1793/60660 Amershi et al. [2014] Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4), 105–120 (2014) https://doi.org/10.1609/aimag.v35i4.2513 Kaufmann et al. [2023] Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Kaufmann, T., Weng, P., Bengs, V., Hüllermeier, E.: A Survey of Reinforcement Learning from Human Feedback (2023). https://arxiv.org/abs/2312.14925 Simard et al. [2017] Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Simard, P.Y., Amershi, S., Chickering, D.M., Pelton, A.E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., Wernsing, J.: Machine Teaching: A New Paradigm for Building Machine Learning Systems (2017). http://arxiv.org/abs/1707.06742 Ramos et al. [2020] Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Ramos, G., Meek, C., Simard, P., Suh, J., Ghorashi, S.: Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35(5-6), 413–451 (2020) https://doi.org/10.1080/07370024.2020.1734931 Mosqueira-Rey et al. [2023] Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Mosqueira-Rey, E., Fernández-Castaño, S., Alonso-Ríos, D., Vázquez-Cano, E., López-Meneses, E.: Gamifying machine teaching: Human-in-the-loop approach for diphthong and hiatus identification in spanish language. Procedia Computer Science 225, 3086–3093 (2023) https://doi.org/10.1016/j.procs.2023.10.302 . 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023) Gunning [2017] Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Gunning, D.: Explainable artificial intelligence (xAI). Technical report, Defense Advanced Research Projects Agency (DARPA) (2017). https://www.darpa.mil/program/explainable-artificial-intelligence Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174156 . https://doi.org/10.1145/3173574.3174156 Guillot Suarez [2022] Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Guillot Suarez, C.: Human-in-the-loop Hyperparameter Tuning of Deep Nets to Improve Explainability of Classifications. Master’s thesis, Aalto University. School of Electrical Engineering (2022). http://urn.fi/URN:NBN:fi:aalto-202205223354 Xu [2019] Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Xu, W.: Toward human-centered AI: A perspective from human-computer interaction. Interactions 26(4), 42–46 (2019) https://doi.org/10.1145/3328485 Choung et al. [2023] Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI & SOCIETY 38(2), 733–745 (2023) https://doi.org/10.1007/s00146-022-01473-4 Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion 58, 82–115 (2020) https://doi.org/10.1016/j.inffus.2019.12.012 Freitas [2014] Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Freitas, A.A.: Comprehensible classification models: a position paper. SIGKDD Explor. Newsl. 15(1), 1–10 (2014) https://doi.org/10.1145/2594473.2594475 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Ribeiro, M.T., Singh, S., Guestrin, C.: Model-Agnostic Interpretability of Machine Learning (2016). https://arxiv.org/abs/1606.05386 Slack et al. [2021] Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Slack, D., Hilgard, A., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: Modeling uncertainty in explainability. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 9391–9404. Curran Associates, Inc., ??? (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf Ho et al. [2021] Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Ho, D.J., Yarlagadda, D.V.K., D’Alfonso, T.M., Hanna, M.G., Grabenstetter, A., Ntiamoah, P., Brogi, E., Tan, L.K., Fuchs, T.J.: Deep multi-magnification networks for multi-class breast cancer image segmentation. Computerized Medical Imaging and Graphics 88, 101866 (2021) https://doi.org/10.1016/j.compmedimag.2021.101866 YILMAZ [2019] YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- YILMAZ, V.: Elastic Deformation on Images (2019). https://towardsdatascience.com/elastic-deformation-on-images-b00c21327372 Chollet [2016] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CoRR abs/1610.02357 (2016) 1610.02357 He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015) 1512.03385 Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, pp. 1135–1144. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778 . https://doi.org/10.1145/2939672.2939778 Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018) https://doi.org/10.1145/3236009 Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. Proceedings of the 31st Int. Conf. on Neural Information Processing Systems. NIPS’17, vol. 30, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Watson et al. [2023] Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Watson, D.S., O’Hara, J., Tax, N., Mudd, R., Guy, I.: Explaining Predictive Uncertainty with Information Theoretic Shapley Values (2023) Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74 Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE Computer Society, Los Alamitos, CA, USA (2016). https://doi.org/10.1109/CVPR.2016.319 . https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.319 Lin et al. [2014] Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Lin, M., Chen, Q., Yan, S.: Network In Network. https://arxiv.org/pdf/1312.4400v3.pdf (2014) Bengio [2012] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Bengio, Y.: Practical recommendations for gradient-based training of deep architectures (2012). http://arxiv.org/abs/1206.5533 Feurer and Hutter [2019] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer, ??? (2019). https://doi.org/10.1007/978-3-030-05318-5_1 . https://doi.org/10.1007/978-3-030-05318-5_1 Bergstra et al. [2011] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, pp. 2546–2554. Curran Associates Inc., Red Hook, NY, USA (2011). https://dl.acm.org/doi/10.5555/2986459.2986743 Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, New York, NY, USA (2006) Chen et al. [2023] Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Chen, Z., Mak, S., Wu, C.F.J.: A hierarchical expected improvement method for Bayesian optimization (2023). https://arxiv.org/pdf/1911.07285.pdf Wu et al. [2019] Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019) https://doi.org/10.11989/JEST.1674-862X.80904120 Nogueira [2014–] Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Nogueira, F.: Bayesian Optimization: Open source constrained global optimization tool for Python (2014–). https://github.com/bayesian-optimization/BayesianOptimization Shahriari et al. [2016] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2016) https://doi.org/10.1109/JPROC.2015.2494218 Brochu et al. [2010] Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Brochu, E., Brochu, T., Freitas, N.: A bayesian interactive optimization approach to procedural animation design. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. SCA ’10, pp. 103–112. Eurographics Association, Goslar, DEU (2010). https://dl.acm.org/doi/abs/10.5555/1921427.1921443 Kim et al. [2017] Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14 Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
- Kim, M., Ding, Y., Malcolm, P., Speeckaert, J., Siviy, C.J., Walsh, C.J., Kuindersma, S.: Human-in-the-loop Bayesian optimization of wearable device parameters. PLOS ONE 12(9) (2017) https://doi.org/10.1371/journal.pone.0184054 . Accessed 2024-03-14
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.